Information processing apparatus, information processing method, and program

ABSTRACT

An information processing apparatus includes an acquisition unit that acquires a free viewpoint video based on at least one image among a plurality of images obtained by imaging an imaging region by a plurality of imaging apparatuses, and a reference video, the free viewpoint video indicating the imaging region in a case in which the imaging region is observed from a specific viewpoint position and a specific visual line direction, and an execution unit that executes a specific process in a case in which a difference degree between the free viewpoint video acquired by the acquisition unit and the reference video acquired by the acquisition unit is equal to or more than a first threshold value.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of InternationalApplication No. PCT/JP2020/028078, filed Jul. 20, 2020, the disclosureof which is incorporated herein by reference in its entirety. Further,this application claims priority under from Japanese Patent ApplicationNo. 2019-138239, filed Jul. 26, 2019, the disclosure of which isincorporated herein by reference in its entirety.

BACKGROUND 1. Technical Field

The technology of the present disclosure relates to an informationprocessing apparatus, an information processing method, and a program.

2. Related Art

JP2019-53738A discloses an information processing apparatus including anacquisition unit that acquires information for specifying a viewpointposition or/and a visual line direction for generating a virtualviewpoint image, and a notification unit that gives notification ofinformation on a quality of the virtual viewpoint image generated basedon the viewpoint position or/and the visual line direction acquired bythe acquisition unit.

SUMMARY

One embodiment according to the technology of the present disclosureprovides an information processing apparatus, an information processingmethod, and a program which can contribute to the acquisition of a freeviewpoint video in which a difference degree from a reference video isless than a first threshold value.

A first aspect according to the technology of the present disclosurerelates to an information processing apparatus including an acquisitionunit that acquires a free viewpoint video based on at least one imageamong a plurality of images obtained by imaging an imaging region by aplurality of imaging apparatuses, and a reference video, the freeviewpoint video indicating the imaging region in a case in which theimaging region is observed from a specific viewpoint position and aspecific visual line direction, and an execution unit that executes aspecific process in a case in which a difference degree between the freeviewpoint video acquired by the acquisition unit and the reference videoacquired by the acquisition unit is equal to or more than a firstthreshold value.

A second aspect according to the technology of the present disclosurerelates to the information processing apparatus according to the firstaspect, in which the specific process is a process including anotification process of giving notification that the difference degreeis equal to or more than the first threshold value.

A third aspect according to the technology of the present disclosurerelates to the information processing apparatus according to the secondaspect, in which the notification process is a process of giving atleast one of visible notification, audible notification, or tactilenotification that the difference degree is equal to or more than thefirst threshold value.

A fourth aspect according to the technology of the present disclosurerelates to the information processing apparatus according to any one ofthe first to third aspects, in which the specific process is a processincluding a process of limiting a range in which the viewpoint positionis able to be indicated to a viewpoint position range determined as arange in which the difference degree is set to be less than the firstthreshold value, and limiting a range in which the visual line directionis able to be indicated to a visual line direction range determined as arange in which the difference degree is set to be less than the firstthreshold value.

A fifth aspect according to the technology of the present disclosurerelates to the information processing apparatus according to any one ofthe first to fourth aspects, in which the specific process is a processincluding a change process of performing, on the free viewpoint video,change for setting the difference degree to be less than the firstthreshold value.

A sixth aspect according to the technology of the present disclosurerelates to the information processing apparatus according to the fifthaspect, in which the change process is a process of changing the freeviewpoint video by changing at least one of the viewpoint position orthe visual line direction.

A seventh aspect according to the technology of the present disclosurerelates to the information processing apparatus according to any one ofthe first to sixth aspects, in which the execution unit executes thespecific process depending on a change in the free viewpoint video.

An eighth aspect according to the technology of the present disclosurerelates to the information processing apparatus according to the seventhaspect, further including a setting unit that selectively sets afollowing mode in which the execution unit executes the specific processwhile following the change in the free viewpoint video, and anon-following mode in which the execution unit does not execute specificprocess while following the change in the free viewpoint video.

A ninth aspect according to the technology of the present disclosurerelates to the information processing apparatus according to the eighthaspect, in which the setting unit sets the following mode in a case inwhich a change amount of the difference degree is less than a secondthreshold value, and sets the non-following mode in a case in which thechange amount is equal to or more than the second threshold value.

A tenth aspect according to the technology of the present disclosurerelates to the information processing apparatus according to the eighthaspect, in which the setting unit sets the following mode in a case inwhich a frequency at which the difference degree changes is less than athird threshold value, and sets the non-following mode in a case inwhich the frequency is equal to or more than the third threshold value.

An eleventh aspect according to the technology of the present disclosurerelates to the information processing apparatus according to any one ofthe first to tenth aspects, further including a control unit thatdisplays the free viewpoint video on a first display device.

A twelfth aspect according to the technology of the present disclosurerelates to the information processing apparatus according to theeleventh aspect, in which the first display device includes a detectionunit that detects a posture of the first display device, and theexecution unit executes the specific process in a case in which theposture detected by the detection unit is a predetermined posture, anddoes not execute the specific process in a case in which the posturedetected by the detection unit is a posture different from thepredetermined posture.

A thirteenth aspect according to the technology of the presentdisclosure relates to the information processing apparatus according tothe eleventh or twelfth aspect, in which the control unit displays thereference video on a second display device different from the firstdisplay device.

A fourteenth aspect according to the technology of the presentdisclosure relates to the information processing apparatus according toany one of the first to thirteenth aspects, in which the execution unitexecutes change in the first threshold value according to instructioninformation for giving an instruction for changing the first thresholdvalue received by a reception unit in a case in which the instructioninformation is received by the reception unit.

A fifteenth aspect according to the technology of the present disclosurerelates to the information processing apparatus according to any one ofthe first to fourteenth aspects, in which the free viewpoint video is avirtual viewpoint video.

A sixteenth aspect according to the technology of the present disclosurerelates to the information processing apparatus according to any one ofthe first to fifteenth aspects, in which the reference video is a livebroadcast video.

A seventeenth aspect according to the technology of the presentdisclosure relates to the information processing apparatus according toany one of the first to sixteenth aspects, in which the reference videois an image obtained by imaging the imaging region by a referenceimaging apparatus.

An eighteenth aspect according to the technology of the presentdisclosure relates to the information processing apparatus according toany one of the first to seventeenth aspects, in which the viewpointposition and the visual line direction are a viewpoint position and avisual line direction as indicated from an outside.

A nineteenth aspect according to the technology of the presentdisclosure relates to the information processing apparatus according toany one of the first to eighteenth aspects, in which the specificprocess is a process that contributes to setting the difference degreeto be less than the first threshold value.

A twentieth aspect according to the technology of the present disclosurerelates to an information processing method including acquiring a freeviewpoint video based on at least one image among a plurality of imagesobtained by imaging an imaging region by a plurality of imagingapparatuses, and a reference video, the free viewpoint video indicatingthe imaging region in a case in which the imaging region is observedfrom a specific viewpoint position and a specific visual line direction,and executing a specific process in a case in which a difference degreebetween the acquired free viewpoint video and the acquired referencevideo is equal to or more than a first threshold value.

A twenty-first aspect according to the technology of the presentdisclosure relates to a program causing a computer to execute a processincluding acquiring a free viewpoint video based on at least one imageamong a plurality of images obtained by imaging an imaging region by aplurality of imaging apparatuses, and a reference video, the freeviewpoint video indicating the imaging region in a case in which theimaging region is observed from a specific viewpoint position and aspecific visual line direction, and executing a specific process in acase in which a difference degree between the acquired free viewpointvideo and the acquired reference video is equal to or more than a firstthreshold value.

BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the technology of the disclosure will bedescribed in detail based on the following figures, wherein:

FIG. 1 is a schematic perspective diagram showing an example of anexternal configuration of an information processing system;

FIG. 2 is a conceptual diagram showing an example of a relationshipbetween an information processing apparatus, a smart device, a receiver,an unmanned aerial vehicle, and an imaging apparatus included in theinformation processing system;

FIG. 3 is a block diagram showing an example of a hardware configurationof an electric system of the information processing apparatus accordingto the embodiment and an example of a relationship between theinformation processing apparatus and peripheral devices thereof;

FIG. 4 is a block diagram showing an example of a hardware configurationof an electric system of the smart device;

FIG. 5 is a conceptual diagram showing an example of an aspect in whichthe smart device according to the embodiment is rotated around a rollaxis;

FIG. 6 is a conceptual diagram showing an example of an aspect in whichthe smart device according to the embodiment is rotated around a yawaxis;

FIG. 7 is a conceptual diagram showing an example of an aspect in whichthe smart device according to the embodiment is rotated around a pitchaxis;

FIG. 8 is a block diagram showing an example of a main function of theinformation processing apparatus;

FIG. 9 is a block diagram showing examples of a reference videogeneration unit and a virtual viewpoint video generation unit realizedby a CPU of the information processing apparatus;

FIG. 10 is a conceptual diagram provided for describing an example of aprocess content of generation of a reference video;

FIG. 11 is a conceptual diagram provided for describing an example of aprocess content of generation of a virtual viewpoint video;

FIG. 12 is a transition diagram showing an example of an aspect in whichthe virtual viewpoint video is changed with change in a viewpointposition and a visual line direction;

FIG. 13 is a conceptual diagram provided for describing an example of aprocess content until the reference video is displayed on the receiverand the virtual viewpoint video is displayed on the smart device;

FIG. 14 is a conceptual diagram showing an example of a relationshipbetween the reference video generation unit, the virtual viewpoint videogeneration unit, an acquisition unit, and an execution unit;

FIG. 15 is a conceptual diagram provided for describing an example of aprocess content of the execution unit;

FIG. 16 is a conceptual diagram showing an example of an aspect in whicha warning message or the like is displayed together with the virtualviewpoint video on a display of the smart device;

FIG. 17 is a screen view showing an example of the virtual viewpointvideo displayed on the display of the smart device in a case in which adifference degree is less than a first threshold value;

FIG. 18 is a conceptual diagram provided for describing an example of aprocess content of a setting unit;

FIG. 19 is a flowchart showing an example of a flow of a display controlprocess;

FIG. 20 is a flowchart showing an example of a flow of a settingprocess;

FIG. 21 is a flowchart showing an example of a flow of informationprocessing;

FIG. 22 is a flowchart showing a modification example of the flow of theinformation processing;

FIG. 23 is a conceptual diagram provided for describing a firstmodification example of the process content of the execution unit;

FIG. 24 is a conceptual diagram showing examples of a viewpoint positionrange and a visual line direction range;

FIG. 25 is a conceptual diagram showing a first modification example ofthe process content of the virtual viewpoint video generation unit;

FIG. 26 is a conceptual diagram showing a second modification example ofthe process content of the virtual viewpoint video generation unit;

FIG. 27 is a conceptual diagram showing an example of an aspect in whicha rectangular frame indicating the viewpoint position range and a guidemessage are displayed on the display of the smart device together withthe virtual viewpoint video;

FIG. 28 is a conceptual diagram showing an example of an aspect in whicha viewpoint position indication is received by a touch panel in therectangular frame shown in FIG. 27;

FIG. 29 is a conceptual diagram showing an example of an aspect in whicha circular frame indicating the visual line direction range and theguide message are displayed on the display of the smart device togetherwith the virtual viewpoint video, and a visual line direction indicationis received by the touch panel toward the inside of the circular frame;

FIG. 30 is a conceptual diagram provided for describing a secondmodification example of the process content of the execution unit;

FIG. 31 is a conceptual diagram showing a third modification example ofthe process content of the virtual viewpoint video generation unit;

FIG. 32 is a conceptual diagram provided for describing a modificationexample of the process content of the setting unit;

FIG. 33 is a conceptual diagram provided for describing a thirdmodification example of the process content of the execution unit;

FIG. 34 is a conceptual diagram provided for describing a fourthmodification example of the process content of the execution unit;

FIG. 35 is a block diagram showing an example of a configuration of thesmart device in a case in which an information processing apparatus sideprogram is stored in a storage of the smart device and an informationprocessing apparatus side process is executed by the CPU of the smartdevice; and

FIG. 36 is a block diagram showing an example of an aspect in which theinformation processing apparatus side program is installed in thecomputer of the information processing apparatus from a storage mediumin which the information processing apparatus side program is stored.

DETAILED DESCRIPTION

An example of an embodiment according to the technology of the presentdisclosure will be described with reference to the accompanyingdrawings.

First, the terms used in the following description will be described.

CPU refers to an abbreviation of “central processing unit”. RAM refersto an abbreviation of “random access memory”. DRAM refers to anabbreviation of “dynamic random access memory”. SRAM refers to anabbreviation of “static random access memory”. ROM refers to anabbreviation of “read only memory”. SSD refers to an abbreviation of“solid state drive”. HDD refers to an abbreviation of “hard disk drive”.EEPROM refers to an abbreviation of “electrically erasable andprogrammable read only memory”. I/F refers to an abbreviation of“interface”. IC refers to an abbreviation of “integrated circuit”. ASICrefers to an abbreviation of “application specific integrated circuit”.PLD refers to an abbreviation of “programmable logic device”. FPGArefers to an abbreviation of “field-programmable gate array”. SoC refersto an abbreviation of “system-on-a-chip”. CMOS refers to an abbreviationof “complementary metal oxide semiconductor”. CCD refers to anabbreviation of “charge coupled device”. EL refers to an abbreviation of“electro-luminescence”. GPU refers to an abbreviation of “graphicsprocessing unit”. LAN refers to an abbreviation of “local area network”.3D refers to an abbreviation of “3 dimension”. USB refers to anabbreviation of “universal serial bus”. In addition, in the descriptionof the present specification, the meaning of “vertical” includes themeaning of a perfect vertical as well as the meaning of a substantiallyvertical including errors allowed in design and manufacturing. Inaddition, in the description of the present specification, the meaningof “match” includes the meaning of a perfect match as well as themeaning of a substantially match including errors allowed in design andmanufacturing.

For example, as shown in FIG. 1, an information processing system 10comprises an information processing apparatus 12, a smart device 14, aplurality of imaging apparatuses 16, an imaging apparatus 18, a wirelesscommunication base station (hereinafter, simply referred to as “basestation”) 20, and a receiver 34. Note that, here, the “smart device 14”refers to a portable multifunctional terminal, such as a smartphone or atablet terminal. Here, the smart device 14 is an example of a “firstdisplay device” according to the technology of the present disclosure,the imaging apparatus 18 is an example of a “first imaging apparatus”and a “second imaging apparatus” according to the technology of thepresent disclosure, and the receiver 34 is an example of a “seconddisplay device” according to the technology of the present disclosure.Note that, here, although the receiver 34 is described as an example,the technology of the present disclosure is not limited to this, and maybe an electronic device with a display (for example, a smart device). Inaddition, the number of the base stations 20 is not limited to one, anda plurality of the base stations 20 may be present. Further, thecommunication standards used in the base station include a wirelesscommunication standard including a Long Term Evolution (LTE) standardand a wireless communication standard including a WiFi (802.11) standardand/or a Bluetooth (registered trademark) standard.

The imaging apparatuses 16 and 18 are devices for imaging having a CMOSimage sensor, and each have an optical zoom function and/or a digitalzoom function. Note that another type of image sensor, such as a CCDimage sensor, may be adopted instead of the CMOS image sensor.Hereinafter, for convenience of description, in a case in which adistinction is not necessary, the imaging apparatus 18 and the pluralityof imaging apparatuses 16, are referred to as “plurality of imagingapparatuses” without reference numeral.

The plurality of imaging apparatuses 16 are installed in a soccerstadium 22. Each of the plurality of imaging apparatuses 16 is disposedso as to surround a soccer field 24, and images a region including thesoccer field 24 as an imaging region. Here, an aspect example isdescribed in which each of the plurality of imaging apparatuses 16 isdisposed so as to surround the soccer field 24. However, the technologyof the present disclosure is not limited to this, and the disposition ofthe plurality of imaging apparatuses 16 is decided depending on avirtual viewpoint video requested to be generated by the viewer 28 orthe like. The plurality of imaging apparatuses 16 may be disposed so asto surround the whole soccer field 24, or the plurality of imagingapparatuses 16 may be disposed so as to surround a specific partthereof. The imaging apparatus 18 is installed in an unmanned typeaerial vehicle (for example, a multi rotorcraft type unmanned aerialvehicle), and images the region including the soccer field 24 as theimaging region in a bird's-eye view from the sky. The imaging region ofthe region including the soccer field 24 in a bird's-eye view from thesky refers to an imaging face on the soccer field 24 by the imagingapparatus 18.

The information processing apparatus 12 is installed in a control room32. The plurality of imaging apparatuses 16 and the informationprocessing apparatus 12 are connected to each other via a LAN cable 30,and the information processing apparatus 12 controls the plurality ofimaging apparatuses 16 and acquires an image obtained by being imaged byeach of the plurality of imaging apparatuses 16. Note that although theconnection using a wired communication method by the LAN cable 30 isdescribed as an example here, the technology of the present disclosureis not limited to this, and the connection using a wirelesscommunication method may be used.

In the soccer stadium 22, spectator seats 26 are provided so as tosurround the soccer field 24, and the viewer 28 sits in the spectatorseat 26. The viewer 28 owns the smart device 14, and the smart device 14is used by the viewer 28. Note that, here, an aspect example isdescribed in which the viewer 28 is present in the soccer stadium 22,but the technology of the present disclosure is not limited to this, andthe viewer 28 may be present outside the soccer stadium 22.

The base station 20 transmits and receives various pieces of informationto and from the information processing apparatus 12 and the unmannedaerial vehicle 27 via radio waves. That is, the information processingapparatus 12 is connected to the unmanned aerial vehicle 27 via the basestation 20 in the wirelessly communicable manner. The informationprocessing apparatus 12 controls the unmanned aerial vehicle 27 bywirelessly communicating with the unmanned aerial vehicle 27 via thebase station 20, and acquires the image obtained by being imaged by theimaging apparatus 18 from the unmanned aerial vehicle 27.

The base station 20 transmits various pieces of information to thereceiver 34 via the wireless communication. The information processingapparatus 12 transmits various videos to the receiver 34 via the basestation 20, the receiver 34 receives various videos transmitted from theinformation processing apparatus 12, and the received various videos aredisplayed on a screen 34A. Note that the receiver 34 is used for viewingby an unspecified number of spectators, for example. A location in whichthe receiver 34 is installed may be inside the soccer stadium 22 oroutside the soccer stadium 22 (for example, a public viewing venue).Note that, here, although an aspect example is described in whichvarious pieces of information is transmitted to the receiver 34 via thewireless communication, the technology of the present disclosure is notlimited to this, and for example, an aspect may be adopted in whichvarious pieces of information is transmitted to the receiver 34 via awired communication.

The information processing apparatus 12 is a device corresponding to aserver, and the smart device 14 is a device corresponding to a clientterminal with respect to the information processing apparatus 12. By theinformation processing apparatus 12 and the smart device 14 wirelesslycommunicating with each other via the base station 20, the smart device14 requests the information processing apparatus 12 to provide variousservices, and the information processing apparatus 12 provides theservices to the smart device 14 in response to the request from thesmart device 14.

For example, as shown in FIG. 2, the information processing apparatus 12acquires a bird's-eye view video 46A showing the region including thesoccer field 24 in a case of being observed from the sky from theunmanned aerial vehicle 27. The bird's-eye view video 46A is a movingimage obtained by imaging the region including the soccer field 24 asthe imaging region (hereinafter, also simply referred to as “imagingregion”) in a bird's-eye view from the sky by the imaging apparatus 18of the unmanned aerial vehicle 27. Note that, here, although a case inwhich the bird's-eye view video 46A is the moving image is described asan example, the bird's-eye view video 46A is not limited to this, andmay be a still image showing the region including the soccer field 24 ina case of being observed from the sky.

The information processing apparatus 12 acquires an imaging video 46Bshowing the imaging region in a case of being observed from eachposition of the plurality of imaging apparatuses 16 from each of theplurality of imaging apparatuses 16. The imaging video 46B is a movingimage obtained by imaging the imaging region by each of the plurality ofimaging apparatuses 16. Note that, here, although a case in which theimaging video 46B is the moving image is described as an example, theimaging video 46B is not limited to this, and may be a still imageshowing the imaging region in a case of being observed from eachposition of the plurality of imaging apparatuses 16.

The information processing apparatus 12 generates a virtual viewpointvideo 46C based on the bird's-eye view video 46A and the imaging video46B. The virtual viewpoint video 46C is a video showing the imagingregion in a case in which the imaging region is observed from a specificviewpoint position and a specific visual line direction. In the exampleshown in FIG. 2, the virtual viewpoint video 46C refers to, for example,the virtual viewpoint video showing the imaging region in a case inwhich the imaging region is observed from a viewpoint position 42 and avisual line direction 44 in a spectator seat 26. Examples of the virtualviewpoint video 46C include a moving image using 3D polygons. Theviewpoint position 42 and the visual line direction 44 are not fixed.That is, the viewpoint position 42 and the visual line direction 44 arechanged in response to an instruction from the viewer 28 or the like.Here, an aspect example is described in which the viewpoint position 42and the visual line direction 44 are not fixed, but the technology ofthe present disclosure is not limited to this, and the viewpointposition 42 and the visual line direction 44 may be fixed.

The information processing apparatus 12 generates the moving image usingthe 3D polygons by composing the plurality of images (here, for example,the bird's-eye view video 46A and the imaging video 46B) obtained byimaging the imaging region by the plurality of imaging apparatuses. Theinformation processing apparatus 12 generates the virtual viewpointvideo corresponding to a case in which the imaging region is observedfrom any position and any direction based on the moving image using thegenerated 3D polygons. In the present embodiment, the informationprocessing apparatus 12 generates the virtual viewpoint video 46 showingthe subject in a case in which the subject is observed from theviewpoint position 42 and the visual line direction 44. Stated anotherway, the virtual viewpoint video 46 refers to the video corresponding tothe video obtained by imaging by a virtual imaging apparatus(hereinafter, also referred to as “virtual imaging apparatus”) installedat the viewpoint position 42 with the visual line direction 44 as theimaging direction. Here, the moving image is described as an example ofthe virtual viewpoint video 46C, but the technology of the presentdisclosure is not limited to this, and a still image may be used. Notethat the viewpoint position 42 is an example of a “specific viewpointposition” according to the technology of the present disclosure, and thevisual line direction 44 is an example of a “specific visual linedirection” according to the technology of the present disclosure. Inaddition, the virtual viewpoint video 46C is an example of a “freeviewpoint video” according to the technology of the present disclosure.

In addition, here, an aspect example is described in which thebird's-eye view video 46A obtained by being imaged by the imagingapparatus 18 is also used for generation, but the technology of thepresent disclosure is not limited to this. For example, the bird's-eyeview video 46A is not provided for generation of the virtual viewpointvideo 46C, and only a plurality of the imaging videos 46B obtained bybeing imaged by the plurality of imaging apparatuses 16 may be used forgenerating the virtual viewpoint video 46C. That is, the virtualviewpoint video 46C may be generated only from the videos obtained bybeing imaged by the plurality of imaging apparatuses 16 without usingthe video obtained by the imaging apparatus 18 (for example, a multirotorcraft type unmanned aerial vehicle). Note that in a case in whichthe video obtained from the imaging apparatus 18 (for example, a multirotorcraft type unmanned aerial vehicle) is used, a more accuratevirtual viewpoint video can be generated.

The information processing apparatus 12 selectively transmits areference video 46B1 (see FIGS. 9 and 10), which will be describedbelow, and the virtual viewpoint video 46C to the smart device 14 andthe receiver 34 as the distribution video. Note that, in the following,for convenience of description, in a case in which a distinction is notnecessary, the reference video 46B1 and the virtual viewpoint video 46Care also referred to as the “distribution video”.

For example, as shown in FIG. 3, the information processing apparatus 12comprises a computer 50, a reception device 52, a display 53, a firstcommunication I/F 54, and a second communication I/F 56. The computer 50comprises a CPU 58, a storage 60, and a memory 62, and the CPU 58, thestorage 60, and the memory 62 are connected to each other via a bus line64. In the example shown in FIG. 3, for convenience of illustration, onebus line is shown as the bus line 64, but a plurality of bus lines maybe used. In addition, the bus line 64 may include a serial bus or aparallel bus configured by a data bus, an address bus, a control bus,and the like.

The CPU 58 controls the whole information processing apparatus 12.Various parameters and various programs are stored in the storage 60.The storage 60 is a non-volatile storage device. Here, a flash memory isadopted as an example of the storage 60, but the technology of thepresent disclosure is not limited to this, and an EEPROM, an HDD, anSSD, or the like may be used. The memory 62 is a storage device. Variouspieces of information are transitorily stored in the memory 62. Thememory 62 is used as a work memory by the CPU 58. Here, a DRAM isadopted as an example of the memory 62, but the technology of thepresent disclosure is not limited to this, and another type of storagedevice may be used.

The reception device 52 receives the instruction from a user or the likeof the information processing apparatus 12. Examples of the receptiondevice 52 include a touch panel, a hard key, and a mouse. The receptiondevice 52 is connected to the bus line 64 and the like, and the CPU 58acquires the instruction received by the reception device 52.

The display 53 is connected to the bus line 64 and displays variouspieces of information under the control of the CPU 58. Examples of thedisplay 53 include a liquid crystal display. Note that another type ofdisplay, such as an organic EL display or an inorganic EL display, maybe adopted as the display 53 without being limited to the liquid crystaldisplay.

The first communication I/F 54 is connected to the LAN cable 30. Thefirst communication I/F 54 is realized by a device having an FPGA, forexample. The first communication I/F 54 is connected to the bus line 64and controls the exchange of various pieces of information between theCPU 58 and the plurality of imaging apparatuses 16. For example, thefirst communication I/F 54 controls the plurality of imaging apparatuses16 in response to the request of the CPU 58. In addition, the firstcommunication I/F 54 acquires the imaging video 46B (see FIG. 2)obtained by being imaged by each of the plurality of imaging apparatuses16, and outputs the acquired imaging video 46B to the CPU 58. Note that,here, although the wired communication I/F is described as an example ofthe first communication I/F 54, a wireless communication I/F, such as ahigh-speed wireless LAN, may be used.

The second communication I/F 56 is connected to the base station 20 inthe wirelessly communicable manner. The second communication I/F 56 isrealized by a device having an FPGA, for example. The secondcommunication I/F 56 is connected to the bus line 64. The secondcommunication I/F 56 controls the exchange of various pieces ofinformation between the CPU 58 and the unmanned aerial vehicle 27 by thewireless communication method via the base station 20. In addition, thesecond communication I/F 56 controls the exchange in various pieces ofinformation between the CPU 58 and the smart device 14 by the wirelesscommunication method via the base station 20. In addition, the secondcommunication I/F 56 controls the transmission of various videos to thereceiver 34 by the CPU 58 by the wireless communication method via thebase station 20. Note that at least one of the first communication I/F54 or the second communication I/F 56 can be configured by a fixedcircuit instead of an FPGA. In addition, at least one of the firstcommunication I/F 54 or the second communication I/F 56 may be a circuitconfigured by an ASIC, an FPGA, and/or a PLD or the like.

For example, as shown in FIG. 4, the smart device 14 comprises acomputer 70, a gyro sensor 74, a reception device 76, a display 78, amicrophone 80, a speaker 82, an imaging apparatus 84, and acommunication I/F 86. The computer 70 comprises a CPU 88, a storage 90,and a memory 92, and the CPU 88, the storage 90, and the memory 92 areconnected to each other via a bus line 94. In the example shown in FIG.4, for convenience of illustration, one bus line is shown as the busline 94. However, the bus line 94 is configured by a serial bus or isconfigured to include a data bus, an address bus, a control bus, and thelike. In addition, in the example shown in FIG. 4, the CPU 88, thereception device 76, the display 78, the microphone 80, the speaker 82,the imaging apparatus 84, and the communication I/F 86 are connected bya common bus, but the CPU 88 and each device may be connected by adedicated bus or a dedicated communication line.

The CPU 88 controls the whole smart device 14. Various parameters andvarious programs are stored in the storage 90. The storage 90 is anon-volatile storage device. Here, a flash memory is adopted as anexample of the storage 90, but the technology of the present disclosureis not limited to this, and an EEPROM, an HDD, an SSD, or the like maybe used. Various pieces of information are transitorily stored in thememory 92, and the memory 92 is used as a work memory by the CPU 88.Here, an RAM is adopted as an example of the memory 92, but thetechnology of the present disclosure is not limited to this, and anothertype of storage device may be used.

The gyro sensor 74 measures an angle around a yaw axis of the smartdevice 14 (hereinafter, also referred to as “yaw angle”), an anglearound a roll axis of the smart device 14 (hereinafter, also referred toas “roll angle”), and an angle around a pitch axis of the smart device14 (hereinafter, also referred to as “pitch angle”). The gyro sensor 74is connected to the bus line 94, and angle information indicating theyaw angle, the roll angle, and the pitch angle measured by the gyrosensor 74 is acquired by the CPU 88 via the bus line 94 and the like.

The reception device 76 is an example of a “reception unit (acceptor)”according to the technology of the present disclosure, and receives theinstruction from the user or the like of the smart device 14 (here, forexample, the viewer 28). Examples of the reception device 76 include atouch panel 76A, and a hard key. The reception device 76 is connected tothe bus line 94, and the CPU 88 acquires the instruction received by thereception device 76.

The display 78 is connected to the bus line 94 and displays variouspieces of information under the control of the CPU 88. Examples of thedisplay 78 include a liquid crystal display. Note that another type ofdisplay, such as an organic EL display, may be adopted as the display 78without being limited to the liquid crystal display.

The smart device 14 comprises a touch panel display, and the touch paneldisplay is realized by the touch panel 76A and the display 78. That is,the touch panel display is formed by superimposing the touch panel 76Aon a display region of the display 78 or by building a touch panelfunction in the display 78 (“in-cell” type).

The microphone 80 converts a collected sound into an electric signal.The microphone 80 is connected to the bus line 94. The CPU 88 acquiresthe electric signal obtained by converting the sound collected by themicrophone 80 via the bus line 94.

The speaker 82 converts the electric signal into the sound. The speaker82 is connected to the bus line 94. The speaker 82 receives the electricsignal output from the CPU 88 via the bus line 94, converts the receivedelectric signal into the sound, and outputs the sound obtained byconverting the electric signal to the outside of the smart device 14.

The imaging apparatus 84 acquires an image showing a subject by imagingthe subject. The imaging apparatus 84 is connected to the bus line 94.The image obtained by imaging the subject by the imaging apparatus 84 isacquired by the CPU 88 via the bus line 94. Here, the speaker 82 isintegrated with the smart device 14, but the sound output by a separateheadphone (including earphones) connected to the smart device 14 by wireor wirelessly may be adopted.

The communication I/F 86 is connected to the base station 20 in thewirelessly communicable manner. The communication I/F 86 is realized by,for example, a device configured by circuits (for example, an ASIC, anFPGA, and/or a PLD). The communication I/F 86 is connected to the busline 94. The communication I/F 86 controls the exchange of variouspieces of information between the CPU 88 and an external device by thewireless communication method via the base station 20. Here, examples ofthe “external device” include the information processing apparatus 12.

For example, as shown in FIG. 5, the roll axis is an axis that passesthrough a center of the display 78 of the smart device 14. A rotationangle of the smart device 14 around the roll axis is measured by thegyro sensor 74 as the roll angle. In addition, as shown in FIG. 6, forexample, the yaw axis is an axis that passes through a center of alateral peripheral surface of the smart device 14 in a longitudinaldirection of the lateral peripheral surfaces. A rotation angle of thesmart device 14 around the yaw axis is measured by the gyro sensor 74 asthe yaw angle. Further, as shown in FIG. 7, for example, the pitch axisis an axis that passes through a center of the lateral peripheralsurface of the smart device 14 in a lateral direction of the lateralperipheral surfaces. The rotation angle of the smart device 14 aroundthe pitch axis is measured by the gyro sensor 74 as the pitch angle.

For example, as shown in FIG. 8, in the information processing apparatus12, the storage 60 stores an information processing program 60A, adisplay control program 60B, and a setting program 60C. Note that, inthe following, in a case in which a distinction is not necessary, theinformation processing program 60A, the display control program 60B, andthe setting program 60C are referred to as an “information processingapparatus side program” without reference numeral.

The CPU 58 is an example of a “processor” according to the technology ofthe present disclosure, and the memory 62 is an example of a “memory”according to the technology of the present disclosure. The CPU 58 readsout the information processing apparatus side program from the storage60, and expands the readout information processing apparatus sideprogram in the memory 62. The CPU 58 performs the exchange in variouspieces of information between the smart device 14, the imaging apparatus16, and the unmanned aerial vehicle 27 according to the informationprocessing apparatus side program expanded in the memory 62, andperforms transmission of various videos to the receiver 34.

The CPU 58 reads out the information processing program 60A from thestorage 60, and expands the readout information processing program 60Ain the memory 62. The CPU 58 is operated as an acquisition unit 102 andan execution unit 104 according to the information processing program60A expanded in the memory 62. The CPU 58 is operated as the acquisitionunit 102 and the execution unit 104 to execute information processing(see FIG. 21), which will be described below.

The CPU 58 reads out the display control program 60B from the storage60, and expands the readout display control program 60B in the memory62. The CPU 58 is operated as a control unit 106 according to thedisplay control program 60B expanded in the memory 62 to execute adisplay control process (see FIG. 19) described below.

The CPU 58 reads out the setting program 60C from the storage 60, andexpands the readout setting program 60C in the memory 62. The CPU 58 isoperated as a setting unit 108 according to the setting program 60Cexpanded in the memory 62. The CPU 58 is operated as the setting unit108 to execute a setting process (see FIG. 20), which will be describedbelow.

For example, as shown in FIG. 9, a video generation program 60D isstored in the storage 60, and the CPU 58 reads out the video generationprogram 60D from the storage 60 and expands the readout video generationprogram 60D in the memory 62. The CPU 58 is operated as a referencevideo generation unit 100A and a virtual viewpoint video generation unit100B according to the video generation program 60D expanded in thememory 62. The reference video generation unit 100A generates thereference video 46B1, and the virtual viewpoint video generation unit100B generates the virtual viewpoint video 46C.

For example, as shown in FIG. 10, in the information processingapparatus 12, imaging video designation information is received by thereception device 52. The imaging video designation information refers toinformation for designating one of a plurality of the imaging videos.The imaging video designation information is received by the receptiondevice 52, for example. The reception of the imaging video designationinformation by the reception device 52 is realized, for example, byoperating the reception device 52 by a video creator (not shown).Specifically, in a state in which the plurality of imaging videos 46Bare displayed side by side on the display 53 of the informationprocessing apparatus 12, one of the plurality of imaging videos isdesignated according to the imaging video designation informationreceived by the reception device 52 by the video creator.

The reference video generation unit 100A generates the reference video46B1 by receiving the plurality of imaging videos from the imagingapparatus 18 and the plurality of imaging apparatuses 16, acquiring theimaging video designated by the imaging video designation informationfrom the received plurality of imaging videos, and performing variouspieces of signal processing (for example, known signal processing) onthe acquired imaging video 46B. That is, the reference video 46B1 is thevideo obtained by imaging the imaging region by an imaging apparatus(hereinafter, also referred to as a “reference imaging apparatus”) ofany of the plurality of imaging apparatuses 16. Here, the “referencevideo 46B1” is, for example, a live broadcast video, a video imaged inadvance, or the like. The live broadcast video or the video imaged inadvance is, for example, a video broadcast on a television ordistributed on the Internet. Note that, here, the live broadcast videois generated based on the imaging video 46B indicated by imaging videoindication information, but it is merely an example, and the imagingvideo 46B itself designated by the imaging video designation informationmay be adopted as the reference video 46B1.

For example, as shown in FIG. 11, the touch panel 76A of the smartdevice 14 receives a viewpoint visual line indication. In this case, forexample, first, the distribution video from the information processingapparatus 12 is received by the smart device 14. Next, the distributionvideo received from the information processing apparatus 12 by the smartdevice 14 is displayed on the display 78. In a state in which thedistribution video is displayed on the display 78, the viewpoint visualline indication is received by the touch panel 76A by operating thetouch panel 76A by the viewer 28.

The viewpoint visual line indication includes a viewpoint positionindication which is an indication of the viewpoint position 42 (see FIG.2) with respect to the imaging region and a visual line directionindication which is an indication of the visual line direction 44 (seeFIG. 2) with respect to the imaging region. In the present embodiment,the viewpoint position indication is first performed, and then thevisual line direction indication is performed. That is, first, theviewpoint position 42 is decided, and then the visual line direction 44is decided.

Examples of the viewpoint position indication include a tap operation onthe touch panel 76A. In addition, the tap operation may be a single tapoperation or a double tap operation. In addition, instead of the tapoperation, a long touch operation may be performed. Examples of thevisual line direction indication include a slide operation. Here, the“slide operation” refers to, for example, an operation of linearlysliding the position in which an indicator (for example, a finger of theviewer 28) is in contact with the touch panel 76A. In this manner, theposition at which the tap operation is performed on the touch panel 76Acorresponds to the viewpoint position 42 with respect to the imagingregion, and the direction in which the slide operation is performed onthe touch panel 76A corresponds to the visual line direction 44 withrespect to the imaging region.

The virtual viewpoint video generation unit 100B generates the virtualviewpoint video 46C by acquiring the viewpoint visual line indicationreceived by the touch panel 76A, and using the plurality of imagingvideos in response to the acquired viewpoint visual line indication.That is, the virtual viewpoint video generation unit 100B generates thevirtual viewpoint video 46C showing the imaging region in a case inwhich the imaging region is observed from the viewpoint position 42 inresponse to the viewpoint position indication and the visual linedirection 44 in response to the visual line direction indication. Thegeneration of the virtual viewpoint video 46C is realized, for example,by generating the 3D polygons based on the plurality of imaging videos.

For example, as shown in FIG. 12, in a case in which the viewpointvisual line indication is changed, that is, in a case in which theviewpoint position 42 and the visual line direction 44 are changed, anaspect of the virtual viewpoint video 46C, that is, a size, a direction,and the like of the subject shown in the virtual viewpoint video 46 arechanged. Note that the enlargement of the virtual viewpoint video 46C isrealized, for example, according to a pinch-out operation on the touchpanel 76A of the smart device 14, and the reduction of the virtualviewpoint video 46C is realized, for example, according to a pinch-inoperation on the touch panel 76A of the smart device 14.

For example, as shown in FIG. 13, the acquisition unit 102 acquires thereference video 46B1 generated by the reference video generation unit100A from the reference video generation unit 100A. In addition, theacquisition unit 102 acquires the virtual viewpoint video 46C generatedby the virtual viewpoint video generation unit 100B from the virtualviewpoint video generation unit 100B.

The control unit 106 includes a reference video control unit 106A and avirtual viewpoint video control unit 106B. The reference video controlunit 106A controls the receiver 34 such that the reference video 46B1 isdisplayed on the screen 34A by transmitting the reference video 46B1acquired by the acquisition unit 102 to the receiver 34, which is adisplay device different from the display 78 of the smart device 14.That is, the receiver 34 receives the reference video 46B1 transmittedfrom the reference video control unit 106A, and displays the receivedreference video 46B1 on the screen 34A.

The virtual viewpoint video control unit 106B controls the smart device14 such that the virtual viewpoint video 46C is displayed on the display78 by transmitting the virtual viewpoint video 46C acquired by theacquisition unit 102 to the smart device 14. That is, the smart device14 receives the virtual viewpoint video 46C transmitted from the virtualviewpoint video control unit 106B, and displays the received virtualviewpoint video 46C on the display 78.

For example, as shown in FIG. 14, the execution unit 104 executes aspecific process based on the reference video 46B1 acquired by theacquisition unit 102 and the virtual viewpoint video 46C acquired by theacquisition unit 102. Here, the specific process is a process performedin a case in which a difference degree between the reference video 46B1acquired by the acquisition unit 102 and the virtual viewpoint video 46Cacquired by the acquisition unit 102 (hereinafter, simply referred to asthe “difference degree”) is equal to or more than a first thresholdvalue. Examples of the specific process include a process thatcontributes to setting of the difference degree to be less than thefirst threshold value. In addition, the execution unit 104 executes thespecific process depending on the change in the virtual viewpoint video46C. That is, the execution unit 104 executes the specific process whilefollowing the change in the virtual viewpoint video 46C.

For example, as shown in FIG. 15, the execution unit 104 includes adifference degree calculation unit 104A, a determination unit 104B, anda notification processing unit 104C. The difference degree calculationunit 104A calculates the difference degree between the reference video46B1 acquired by the acquisition unit 102 and the virtual viewpointvideo 46C acquired by the acquisition unit 102. The difference degreerefers to, for example, a degree of difference between the referencevideo 46B1 and the virtual viewpoint video 46C. The calculation of thedifference degree is realized, for example, by comparing the featureamounts obtained by performing pattern matching or Fourier transform.

The difference degree calculation unit 104A outputs difference degreeinformation indicating the calculated difference degree to thedetermination unit 104B. The determination unit 104B determines whetheror not the difference degree indicated by the difference degreeinformation input from the difference degree calculation unit 104A isequal to or more than the first threshold value. Examples of a case inwhich the difference degree is equal to or more than the first thresholdvalue include a case in which the match location between the referencevideo 46B1 and the virtual viewpoint video 46C is less than 30%.

Note that in the present embodiment, as the first threshold value, afixed value derived in advance by a sensory test and/or a computersimulation or the like is adopted. The first threshold value does nothave to be the fixed value, and may be, for example, a variable valuethat is changed in response to the instructions received by thereception device 52 (see FIG. 2) and/or the reception device 76.

The determination unit 104B compares the difference degree with thefirst threshold value, and in a case in which the difference degree isequal to or more than the first threshold value, outputs largedifference degree information indicating that the difference degree isequal to or more than the first threshold value to the notificationprocessing unit 104C. Alternatively, the determination unit 104Bcompares the difference degree with the first threshold value, and in acase in which the difference degree is less than the first thresholdvalue, does not output the large difference degree information and waitsfor input of the difference degree information from the differencedegree calculation unit 104A.

When the large difference degree information is input from thedetermination unit 104B, the notification processing unit 104C executesa process including a notification process as the specific processdescribed above. Here, the “notification process” refers to a process ofgiving notification that the difference degree is equal to or more thanthe first threshold value. A notification destination is, for example,the viewer 28. The notification process is realized by transmittingwarning indication information to the smart device 14 by thenotification processing unit 104C. Here, the warning indicationinformation refers to information for instructing the smart device 14 togive a warning to the viewer 28 that the difference degree is equal toor more than the first threshold value. In the example shown in FIG. 15,an aspect is described in which the warning indication information isinput to the smart device 14 in a state in which the virtual viewpointvideo 46C is displayed on the display 78 of the smart device 14.

For example, as shown in FIG. 16, in a case in which the warningindication information is input to the smart device 14, under thecontrol of the CPU 88, a warning message is displayed in a state inwhich the virtual viewpoint video 46C is displayed on the display 78 andthe warning message is parallel to the virtual viewpoint video 46C. Notethat the display of the warning message on the display 78 is an exampleof “visible notification” according to the technology of the presentdisclosure.

In the example shown in FIG. 16, a message “It is quite moved away fromthe live broadcast video!” is shown as the warning message. Note thatthe warning message may be a message other than the above, for example,a message such as “Currently, a video completely different from the livebroadcast video is being displayed.” or “Since a video completelydifferent from the live broadcast video is being displayed, in a case inwhich you desires to bring the currently displayed video closer to thelive broadcast video, please newly indicate of the viewpoint positionand the visual line direction.” may be used. That is, the message needonly be the message that allows the viewer 28 to visually recognize thatthe difference degree between the video (in the example shown in FIG.16, the virtual viewpoint video 46C) that is currently visuallyrecognized by the viewer 28 via the display 78 of the smart device 14and the reference video 46B1 (for example, the live broadcast video)displayed on the screen 34A of the receiver 34 is equal to or more thanthe first threshold value.

In addition, in the example shown in FIG. 16, recommendation viewpointposition information and a recommendation visual line directioninformation are displayed on the display 78 together with the warningmessage. The recommendation viewpoint position refers to the viewpointposition 42 required to reduce the difference degree. The recommendationvisual line direction refers to the visual line direction 44 required toreduce the difference degree. In the example shown in FIG. 16,information including an arrow P1 indicating the recommendationviewpoint position and a guide message “recommendation viewpointposition” is shown as the recommendation viewpoint position information,and information including an arrow P2 indicating the recommendationvisual line direction and a guide message “recommendation visual linedirection” is shown as the recommendation visual line directioninformation. The viewer 28 moves the screen by performing the slideoperation, a flick operation, or the like on the touch panel 76A toadjust the viewpoint position 42.

Note that the CPU 88 of the smart device 14 may control therecommendation viewpoint position information such that the arrow P1 isshorter as the viewpoint position 42 approaches the recommendationviewpoint position and the arrow P1 is longer as the viewpoint position42 is moved away from the recommendation viewpoint position. The CPU 88may also change the direction of the arrow P2 depending on thedifference degree between the current visual line direction 44 and therecommendation visual line direction.

In addition, here, although the aspect example has been described inwhich the message is displayed on the display 78, the visiblenotification that the difference degree is equal to or more than thefirst threshold value may be given by turning on a specific light source(not shown) having a predetermined color (for example, red). Inaddition, the recommendation viewpoint position and the recommendationvisual line direction may be visually recognized by the viewer 28 byselectively turning on and off the light source having a plurality ofcolors (for example, a green light source and a yellow light source) andlengthening or shortening a turning-on/off interval.

In addition, examples of a unit that allows the viewer 28 to perceivethe information corresponding to the warning message described above,the recommendation viewpoint position information, and/or therecommendation visual line direction information include a voicereproduction device and/or a vibrator. In this case, for example, theviewer 28 perceives the information corresponding to the warningmessage, the recommendation viewpoint position information, and/or therecommendation visual line direction information by audible notificationby a voice by using the voice reproduction device and/or tactilenotification by using the vibrator. The audible notification by thevoice may be audible notification by a bone conduction method. Thetactile notification by using the vibrator is realized, for example, bychanging a vibration pattern (for example, a vibration cycle and anamplitude) by the vibrator. As described above, the informationcorresponding to the warning message described above, the recommendationviewpoint position information, and/or the recommendation visual linedirection information need only be perceived by the viewer 28 by thevisible notification by the display 78 or the like, the audiblenotification by using the voice reproduction device, and/or the tactilenotification by the vibrator or the like.

The determination unit 104B compares the difference degree with thefirst threshold value, and in a case in which the difference degree isless than the first threshold value, does not output the largedifference degree information. Therefore, as shown in FIG. 17, thevirtual viewpoint video 46C is displayed on the display 78 of the smartdevice 14 under the control of the CPU 88, but the warning message, therecommendation viewpoint position information, and the recommendationvisual line direction information are not displayed.

For example, as shown in FIG. 18, the setting unit 108 includes adifference degree change amount calculation unit 108A, a change amountdetermination unit 108B, and a mode indication unit 108C. The settingunit 108 selectively sets a following mode and a non-following mode. Thefollowing mode refers to an operation mode in which the execution unit104 executes the notification process while following the change in thevirtual viewpoint video 46C, and the non-following mode refers to anoperation mode in which the execution unit 104 does not execute thenotification process while following the change in the virtual viewpointvideo 46C.

The difference degree change amount calculation unit 108A acquires thedifference degree information from the difference degree calculationunit 104A. That is, the difference degree information is acquired insynchronization with an acquisition timing of the difference degreeinformation by the determination unit 104B (see FIG. 15), and the changeamount of the difference degree is calculated based on the acquireddifference degree information. Specifically, the difference degreechange amount calculation unit 108A first acquires the difference degreeinformation at different timings. Here, in a case in which N is aninteger of 1 or more, the different timings refer to, for example, anNth acquisition timing and an N+1th acquisition timing. Next, thedifference degree change amount calculation unit 108A calculates thechange amount of the difference degree indicated by the differencedegree information acquired at the Nth acquisition timing to thedifference degree indicated by the difference degree informationacquired at the N+1th acquisition timing. Here, examples of the “changeamount” include an absolute value of the difference between thedifference degree indicated by the difference degree informationacquired at the Nth acquisition timing and the difference degreeindicated by the difference degree information acquired at the N+1thacquisition timing. Note that instead of the absolute value of thedifference, a ratio of the difference degree indicated by the differencedegree information acquired at the N+1th acquisition timing to thedifference degree indicated by the difference degree informationacquired at the Nth acquisition timing may be used, or any informationmay be used as long as the information indicates the change amount ofthe difference degree.

The change amount determination unit 108B determines whether or not thechange amount calculated by the difference degree change amountcalculation unit 108A is less than a second threshold value. Note thatin the present embodiment, as the second threshold value, a fixed valuederived in advance by a sensory test and/or a computer simulation or thelike is adopted. The second threshold value does not have to be thefixed value, and may be, for example, a variable value that is changedin response to the instructions received by the reception device 52 (seeFIG. 2) and/or the reception device 76.

The change amount determination unit 108B outputs the informationindicating that the change amount is less than the second thresholdvalue to the mode indication unit 108C in a case in which the changeamount calculated by the difference degree change amount calculationunit 108A is less than the second threshold value. In addition, thechange amount determination unit 108B outputs the information indicatingthat the change amount is equal to or more than the second thresholdvalue to the mode indication unit 108C in a case in which the changeamount calculated by the difference degree change amount calculationunit 108A is equal to or more than the second threshold value.

The mode indication unit 108C outputs following mode indicationinformation for indicating the following mode as the operation mode ofthe execution unit 104 in a case in which the information indicatingthat the change amount is less than the second threshold value is inputfrom the change amount determination unit 108B. The mode indication unit108C outputs non-following mode indication information for indicatingthe non-following mode as the operation mode of the execution unit 104in a case in which the information indicating that the change amount isequal to or more than the second threshold value is input from thechange amount determination unit 108B. In a case in which the followingmode indication information is input from the mode indication unit 108C,the execution unit 104 is operated in the following mode as indicated bythe following mode indication information. In a case in which thenon-following mode indication information is input from the modeindication unit 108C, the execution unit 104 is operated in thenon-following mode as indicated by the non-following mode indicationinformation.

In this way, the setting unit 108 sets the following mode as theoperation mode of the execution unit 104 in a case in which the changeamount of the difference degree is less than the second threshold value,and sets the non-following mode as the operation mode of the executionunit 104 in a case in which the change amount of the difference degreeis equal to or more than the second threshold value.

Next, an operation of the information processing system 10 will bedescribed.

First, an example of a flow of the display control process executed bythe CPU 58 of the information processing apparatus 12 will be describedwith reference to FIG. 19. Note that, here, for convenience ofdescription, the description will be made on the premise that thereference video 46B1 is already generated by the reference videogeneration unit 100A and the virtual viewpoint video 46C is alreadygenerated by the virtual viewpoint video generation unit 100B.

In the display control process shown in FIG. 19, first, in step ST200,the acquisition unit 102 acquires the reference video 46B1 generated bythe reference video generation unit 100A from the reference videogeneration unit 100A, and then the display control process proceeds tostep ST202.

In step ST202, the reference video control unit 106A displays thereference video 46B1 on the receiver 34 by transmitting the referencevideo 46B1 acquired in step ST200 to the receiver 34. That is, thereceiver 34 receives the reference video 46B1, and displays the receivedreference video 46B1 on the screen 34A.

In subsequent step ST204, the acquisition unit 102 acquires the virtualviewpoint video 46C generated by the virtual viewpoint video generationunit 100B from the virtual viewpoint video generation unit 100B, andthen the display control process proceeds to step ST206.

In step ST206, the virtual viewpoint video control unit 106B displaysthe virtual viewpoint video 46C on the smart device 14 by transmittingthe virtual viewpoint video 46C acquired in step ST204 to the smartdevice 14. That is, the smart device 14 receives the virtual viewpointvideo 46C, and displays the received virtual viewpoint video 46C on thedisplay 78.

In subsequent step ST208, the control unit 106 determines whether or nota condition for terminating the display control process (hereinafter,referred to as a “display control process termination condition”) issatisfied. Examples of the display control process termination conditioninclude a condition that an instruction for terminating the displaycontrol process is received by the reception device 52 or 76. In a casein which the display control process termination condition is notsatisfied in step ST208, a negative determination is made, and thedisplay control process proceeds to step ST200. In a case in which thedisplay control process termination condition is satisfied in stepST208, a positive determination is made, and the display control processis terminated.

Next, an example of a flow of the setting process executed by the CPU 58of the information processing apparatus 12 will be described withreference to FIG. 20. Note that, here, for convenience of description,the description will be made on the premise that the difference degreeis calculated by the difference degree calculation unit 104A, and thecalculated difference degree information is output in synchronizationwith the determination unit 104B and the difference degree change amountcalculation unit 108A.

In the setting process shown in FIG. 20, first, in step ST250, thedifference degree change amount calculation unit 108A acquires thedifference degree information from the difference degree calculationunit 104A at different timings, and then the setting process proceeds tostep ST252.

In step ST252, the difference degree change amount calculation unit 108Acalculates the change amount for each difference degree indicated byeach difference degree information acquired at different timings in stepST250, and then the setting process proceeds to step ST254.

In step ST254, the change amount determination unit 108B determineswhether or not the change amount calculated in step ST252 is less thanthe second threshold value. In step ST254, in a case in which the changeamount calculated in step ST252 is equal to or more than the secondthreshold value, a negative determination is made, and the settingprocess proceeds to step ST256. In step ST254, in a case in which thechange amount calculated in step ST252 is less than the second thresholdvalue, a positive determination is made, and the setting processproceeds to step ST258.

In step ST256, the mode indication unit 108C outputs the non-followingmode indication information to the execution unit 104, and then thesetting process proceeds to step ST260. By executing the process of stepST256, the execution unit 104 is operated in the non-following mode asindicated by the non-following mode indication information input fromthe mode indication unit 108C.

In step ST258, the mode indication unit 108C outputs the following modeindication information to the execution unit 104, and then the settingprocess proceeds to step ST260. By executing the process of step ST258,the execution unit 104 is operated in the following mode as indicated bythe following mode indication information input from the mode indicationunit 108C.

In subsequent step ST260, the setting unit 108 determines whether or nota condition for terminating the setting process (hereinafter, referredto as a “setting process termination condition”) is satisfied. Examplesof the setting process termination condition include a condition thatthe reception device 52 or 76 receives an instruction for terminatingthe setting process. In a case in which the setting process terminationcondition is not satisfied in step ST260, a negative determination ismade, and the setting process proceeds to step ST250. In a case in whichthe setting process termination condition is satisfied in step ST260, apositive determination is made, and the setting process is terminated.

Next, an example of a flow of the information processing executed by theCPU 58 of the information processing apparatus 12 will be described withreference to FIG. 21. Note that, here, for convenience of description,the description will be made on the premise that the reference video46B1 is already generated by the reference video generation unit 100Aand the virtual viewpoint video 46C is already generated by the virtualviewpoint video generation unit 100B. In addition, here, the descriptionwill be made on the premise that the virtual viewpoint video 46Cgenerated by the virtual viewpoint video generation unit 100B isdisplayed on the display 78 of the smart device 14. Further, here, thedescription will be made on the premise that the following mode or thenon-following mode is set as the operation mode for the execution unit104.

In the information processing shown in FIG. 21, first, in step ST300,the acquisition unit 102 acquires the reference video 46B1 from thereference video generation unit 100A, and then the informationprocessing proceeds to step ST302.

In step ST302, the acquisition unit 102 acquires the virtual viewpointvideo 46C from the virtual viewpoint video generation unit 100B, andthen the information processing proceeds to step ST304.

In step ST304, the difference degree calculation unit 104A calculatesthe difference degree between the reference video 46B1 acquired in stepST300 and the virtual viewpoint video 46C acquired in step ST302, andthen the information processing proceeds to step ST306.

In step ST306, the determination unit 104B determines whether or not theoperation mode of the execution unit 104 is the following mode. In stepST306, in a case in which the operation mode of the execution unit 104is the non-following mode, a negative determination is made, and theinformation processing proceeds to step ST312. In step ST306, in a casein which the operation mode of the execution unit 104 is the followingmode, a positive determination is made, and the information processingproceeds to step ST308.

In step ST308, the determination unit 104B determines whether or not thedifference degree calculated in step ST304 is equal to or more than thefirst threshold value. In a case in which the difference degree is lessthan the first threshold value in step ST308, a negative determinationis made, and the information processing proceeds to step ST312. In acase in which the difference degree is equal to or more than the firstthreshold value in step ST308, a positive determination is made, and theinformation processing proceeds to step ST310.

In step ST310, the notification processing unit 104C executes thenotification process, and then the information processing proceeds tostep ST312. By executing the notification process by the notificationprocessing unit 104C, warning information is transmitted to the smartdevice 14 by the notification processing unit 104C. In a case in whichthe warning information is received, the smart device 14 displays thewarning message, the recommendation viewpoint position information, andthe recommendation visual line direction information on the display 78together with the virtual viewpoint video 46C. In this way, the visiblenotification of the warning message, the recommendation viewpointposition information, and the recommendation visual line directioninformation is given to notify the viewer 28 that the difference degreeis equal to or more than the first threshold value.

In subsequent step ST312, the execution unit 104 determines whether ornot a condition for terminating the information processing (hereinafter,referred to as an “information processing termination condition”) issatisfied. Examples of the information processing termination conditioninclude a condition that the reception device 52 or 76 receives aninstruction for terminating the information processing. In a case inwhich the information processing termination condition is not satisfiedin step ST312, a negative determination is made, and the informationprocessing proceeds to step ST300. In a case in which the informationprocessing termination condition is satisfied in step ST312, a positivedetermination is made, and the information processing is terminated.

As described above, in the information processing system 10, theacquisition unit 102 acquires the virtual viewpoint video 46C and thereference video 46B1, and the execution unit 104 executes the specificprocess that contributes to setting of the difference degree to be lessthan the first threshold value in a case in which the difference degreebetween the virtual viewpoint video 46C and the reference video 46B1 isequal to or more than the first threshold value. Therefore, it ispossible to contribute to the acquisition of the virtual viewpoint video46C in which the difference degree from the reference video 46B1 is lessthan the first threshold value.

Note that, in the above, the virtual viewpoint video 46C has beendescribed as an example, but instead of the virtual viewpoint video 46C,the imaging video 46B itself may be used as a time slice free viewpointvideo, or the virtual viewpoint video 46C and the time slice freeviewpoint video may be used in combination. In addition, not limited tothe moving image, the free viewpoint video of the still image may beused. As a result, it is possible to contribute to the acquisition ofthe free viewpoint video in which the difference degree from thereference video 46B1 is less than the first threshold value.

In addition, in the information processing system 10, the execution unit104 executes the notification process as the specific process. Byexecuting the notification process, the viewer 28 is notified of thewarning message indicating that the difference degree is equal to ormore than the first threshold value. Therefore, the viewer 28 can bemade aware that the difference degree between the virtual viewpointvideo 46C and the reference video 46B1 is equal to or more than thefirst threshold value.

In addition, the information processing system 10 executes thenotification process in which the viewer 28 is notified of the warningmessage by performing the visible notification, the audiblenotification, and/or the tactile notification. Therefore, the viewer 28can be made perceive that the difference degree between the virtualviewpoint video 46C and the reference video 46B1 is equal to or morethan the first threshold value.

In addition, in the information processing system 10, the execution unit104 executes the specific process depending on the change in the virtualviewpoint video 46C. Therefore, even in a case in which the virtualviewpoint video 46C is changed, it can immediately contribute to theacquisition of the virtual viewpoint video of which the differencedegree with the reference video 46B1 is less than the first thresholdvalue.

In addition, in the information processing system 10, the following modeand the non-following mode are selectively set by the setting unit 108.Therefore, it is possible to reduce the power consumption as compared toa case in which the specific process is always executed while followingthe change in the virtual viewpoint video 46C.

In addition, in the information processing system 10, the setting unit108 sets the following mode in a case in which the change amount of thedifference degree is less than the second threshold value, and sets thenon-following mode in a case in which the change amount of thedifference degree is equal to or more than the second threshold value.Therefore, the specific process can be prevented from being executedcontrary to the intention of the viewer 28.

In addition, in the information processing system 10, the control unit106 displays the virtual viewpoint video 46C on the display 78 of thesmart device 14. Therefore, it is possible for the viewer 28 to visuallyrecognize the virtual viewpoint video 46C.

In addition, in the information processing system 10, the control unit106 displays the reference video 46B1 on the screen 34A of the receiver34. Therefore, it is possible for the viewer of the receiver 34 tovisually recognize the reference video 46B1.

In addition, in the information processing system 10, the live broadcastvideo is adopted as the reference video 46B1. Therefore, it is possibleto contribute to the acquisition of the virtual viewpoint video 46C inwhich the difference degree from the live broadcast video is less thanthe first threshold value.

In addition, in the information processing system 10, the video obtainedby imaging the imaging region by the reference imaging apparatus is usedas the reference video 46B1. Therefore, as compared to a case in which avideo other than the video obtained by imaging the imaging region (forexample, a virtual video) is used as the reference video, it is possiblefor the viewer 28 or the like to easily grasp the difference between thevideo obtained by imaging and the virtual viewpoint video 46C.

In addition, in the information processing system 10, the viewpointposition 42 and the visual line direction 44 as indicated from theoutside are used. That is, the viewpoint position 42 and the visual linedirection 44 are decided in response to the viewpoint visual lineindication received by the touch panel 76A. Therefore, it is possible togenerate the virtual viewpoint video 46C in a case in which the imagingregion is observed from the viewpoint position and the visual linedirection intended by the viewer 28.

Further, in the information processing system 10, as the specificprocess, the process that contributes to setting the difference degreeto be less than the first threshold value is adopted. Therefore, it ispossible to contribute to the acquisition of the virtual viewpoint video46C in which the difference degree from the reference video 46B1 is lessthan the first threshold value.

Note that in the embodiment described above, the aspect example has beendescribed in which the reference video 46B1 and the virtual viewpointvideo 46C are generated by the information processing apparatus 12, butthe technology of the present disclosure is not limited to this. Thereference video 46B1 and/or the virtual viewpoint video 46C may begenerated by a device different from the information processingapparatus 12 (hereinafter, referred to as a “video generation device”),and the video generated by the video generation device may be acquiredby the acquisition unit 102.

In the embodiment described above, the notification process has beendescribed as an example of the specific process executed by theexecution unit 104, but the technology of the present disclosure is notlimited to this, and the information processing shown in FIG. 22 may beexecuted instead of the information processing (see FIG. 21) describedin the embodiment. The information processing shown in FIG. 22 isdifferent from the information processing shown in FIG. 21 in that stepST310A is provided instead of step ST310. In step ST310A, a limitationprocessing unit 104D (see FIG. 23) executes a limitation process.

The limitation process is a process for limiting a range in which theviewpoint position 42 can be indicated to a viewpoint position range R1(see FIG. 24) determined as a range in which the difference degree isset to be less than the first threshold value and limiting a range inwhich the visual line direction 44 can be indicated to a visual linedirection range R2 (see FIG. 24) determined as the range in which thedifference degree is set to be less than the first threshold value.

For example, as shown in FIG. 23, the limitation processing unit 104Doutputs viewpoint visual line limitation instruction information to thevirtual viewpoint video generation unit 100B by executing the limitationprocess. The viewpoint visual line limitation instruction information isinformation including viewpoint position limitation instructioninformation and visual line direction limitation instructioninformation. The viewpoint position limitation instruction informationrefers to information indicating an instruction for limiting the rangein which the viewpoint position 42 can be indicated to the viewpointposition range R1 (see FIG. 24), and the visual line directionlimitation instruction information refers to information indicating aninstruction for limiting the range in which the visual line direction 44can be indicated to the visual line direction range R2 (see FIG. 24).

In this case, for example, as shown in FIG. 24, the viewpoint position42 is forcibly set in the viewpoint position range R1 and the visualline direction 44 is forcibly set in the visual line direction range R2.Stated another way, the viewpoint position 42 follows the viewpointposition range R1, and the visual line direction 44 follows the visualline direction range R2. For example, as shown by a two dot chain linein FIG. 24, even in a case in which the viewpoint position 42 and thevisual line direction 44 are once moved away from the viewpoint positionrange and the visual line direction range, the viewpoint position 42 andthe visual line direction 44 are returned again in the viewpointposition range and the visual line direction range.

The disposition position of the viewpoint position 42 in the viewpointposition range R1 is decided depending on a positional relationshipbetween the viewpoint position range R1 and the current viewpointposition 42. For example, in the viewpoint position range R1, theviewpoint position 42 is changed to a position closest to the currentviewpoint position 42. In addition, the direction of the visual linedirection in the visual line direction range R2 is decided depending ona positional relationship between the visual line direction range R2 andthe current visual line direction 44. For example, in the visual linedirection range R2, the visual line direction 44 is changed to adirection closest to the current visual line direction 44.

For example, as shown in FIG. 25, the virtual viewpoint video generationunit 100B includes a viewpoint visual line range determination unit100B1 and a video generation execution unit 100B2. The viewpoint visualline range determination unit 100B1 acquires the viewpoint visual lineindication from the smart device 14, and also acquires the viewpointvisual line limitation instruction information from the limitationprocessing unit 104D.

The viewpoint visual line range determination unit 100B1 determineswhether or not the viewpoint position 42 is in the viewpoint positionrange R1 and the visual line direction 44 is in the visual linedirection range R2 based on the viewpoint visual line indication and theviewpoint visual line limitation indication information. In a case inwhich the viewpoint position 42 is outside the viewpoint position rangeR1 and the visual line direction 44 is outside the visual line directionrange R2, the viewpoint visual line range determination unit 100B1adjusts the viewpoint visual line indication such that the currentviewpoint position 42 is set in the viewpoint position range R1 and thecurrent visual line direction 44 is set in the visual line directionrange R2. Moreover, the viewpoint visual line range determination unit100B1 outputs the adjusted viewpoint visual line indication to the videogeneration execution unit 100B2. The video generation execution unit100B2 generates the virtual viewpoint video 46C in response to theviewpoint visual line indication input from the viewpoint visual linerange determination unit 100B1. As a result, the virtual viewpoint video46C of which the difference degree with the reference video 46B1 is lessthan the first threshold value is generated depending on the viewpointposition 42 in the viewpoint position range R1 and the visual linedirection 44 in the visual line direction range R2. Therefore, it ispossible to maintain the difference degree between the reference video46B1 and the virtual viewpoint video 46C to be less than the firstthreshold value.

In addition, for example, as shown in FIG. 26, the limitation processingunit 104D may output the viewpoint visual line limitation instructioninformation to the smart device 14 in addition to the viewpoint visualline range determination unit 100B1. In this case, for example, as shownin FIG. 27, in the smart device 14, under the control of the CPU 88, arectangular frame R1A indicating an outer edge of the viewpoint positionrange R1 superimposed on the virtual viewpoint video 46C is displayedthe display 78 and a guide message “Please indicate the viewpointposition in the rectangular frame.” is displayed. For example, as shownin FIG. 28, in a case in which the viewpoint position indication fromthe viewer 28 is received by the touch panel 76A in the rectangularframe R1A, the viewpoint position 42 is changed to a position in theviewpoint position range R1.

In addition, in a case in which the viewpoint position indication isreceived by the touch panel 76A, the rectangular frame R1A is deletedfrom the display 78. Moreover, subsequently, for example, as shown inFIG. 29, in the smart device 14, under the control of the CPU 88, acircular frame R2A indicating an outer edge of the visual line directionrange R2 superimposed on the virtual viewpoint video 46C is displayed onthe display 78 and a guide message “Please indicate the visual linedirection toward an inside of the circular frame.” is displayed. In thiscase, for example, in a case in which the visual line directionindication (for example, the slide operation) from the viewer 28 isreceived by the touch panel 76A toward the circular frame R2A, thevisual line direction 44 is changed to a direction in the visual linedirection range R2.

In this way, in a case in which the viewpoint position 42 is changed tothe position in the viewpoint position range R1 and the visual linedirection 44 is changed to the direction in the visual line directionrange R2, the virtual viewpoint video 46C is generated by the virtualviewpoint video generation unit 100B depending on the changed viewpointposition 42 and the changed visual line direction 44.

In addition, in the example shown in FIG. 23, an aspect example isdescribed in which the limitation process is executed by the limitationprocessing unit 104D, but the technology of the present disclosure isnot limited to this. For example, as shown in FIG. 30, the executionunit 104 may include a change processing unit 104E instead of thelimitation processing unit 104D. In the example shown in FIG. 23, thelimitation processing unit 104D is not shown, but the limitationprocessing unit 104D can be present together with the change processingunit 104E.

The change processing unit 104E executes a change process. The changeprocess refers to a process of performing, on the virtual viewpointvideo 46C, change for setting the difference degree to be less than thefirst threshold value. That is, in a case in which the change processingunit 104E executes the change process, the virtual viewpoint video 46Cis changed such that the difference degree is set to be less than thefirst threshold value. In a case in which the large difference degreeinformation is input from the determination unit 104B, the changeprocessing unit 104E generates changeable amount information based onthe difference degree information input from the difference degreecalculation unit 104A via the determination unit 104B, and outputs thegenerated changeable amount information to the virtual viewpoint videogeneration unit 100B. The changeable amount information refers to alimit value of the change amount required to change the virtualviewpoint video 46C, that is, an upper limit value of the change amountrequired to set the difference degree indicated by the difference degreeinformation to be less than the first threshold value (hereinafter,referred to as a “changeable amount”).

For example, as shown in FIG. 31, the virtual viewpoint video generationunit 100B includes the video generation execution unit 100B2 and aviewpoint visual line indication adjustment unit 100B3. The viewpointvisual line indication adjustment unit 100B3 acquires the viewpointvisual line indication from the smart device 14 and acquires thechangeable amount information from the change processing unit 104E.

The viewpoint visual line indication adjustment unit 100B3 changes theviewpoint position 42 and the visual line direction 44 by adjusting theviewpoint visual line indication. Specifically, the viewpoint visualline indication adjustment unit 100B3 adjusts the viewpoint visual lineindication in the changeable amount indicated by the changeable amountinformation. Here, the adjustment of the viewpoint visual lineindication refers to the adjustment of the viewpoint position indicationand the visual line direction indication. The viewpoint visual lineindication adjustment unit 100B3 outputs the adjusted viewpoint visualline indication to the video generation execution unit 100B2. The videogeneration execution unit 100B2 generates the virtual viewpoint video46C by using the plurality of imaging videos in response to theviewpoint visual line indication input from the viewpoint visual lineindication adjustment unit 100B3.

As a result, the virtual viewpoint video 46C is changed by the changeamount required to set the difference degree to be less than the firstthreshold value, so that the difference degree between the virtualviewpoint video 46C and the reference video 46B1 can be maintained to beless than the first threshold value. In addition, since the viewpointposition indication and the visual line direction indication areadjusted, it is possible to easily change the virtual viewpoint video46C of which the difference degree with the reference video 46B1 is setto be less than the first threshold value as compared to a case in whichneither the viewpoint position indication nor the visual line directionindication is adjusted.

Note that, here, although the aspect example has been described in whichboth the viewpoint position indication and the visual line directionindication are adjusted, the technology of the present disclosure is notlimited to this, and any one of the viewpoint position indication or thevisual line direction indication may be adjusted by the viewpoint visualline indication adjustment unit 100B3 such that the difference degree isset to be less than the first threshold value.

In addition, in the example shown in FIG. 18, the change amountdetermination unit 108B has been described as an example, but thetechnology of the present disclosure is not limited to this. Forexample, as shown in FIG. 32, the setting unit 108 may include a changefrequency determination unit 108B1 instead of the change amountdetermination unit 108B. The change frequency determination unit 108B1determines whether or not a frequency at which the difference degreechanges is less than a third threshold value. Here, examples of the“frequency at which the difference degree changes” include a frequencyat which the difference degree exceeds the first threshold valuedescribed (for example, the frequency per unit time). The changefrequency determination unit 108B1 outputs information indicating thatthe frequency is less than the third threshold value to the modeindication unit 108C in a case in which the frequency at which thedifference degree changes is less than the third threshold value, andoutputs information indicating that the frequency is equal to or morethan the third threshold value to the mode indication unit 108C in acase in which the frequency at which the difference degree changes isequal to or more than the third threshold value. As a result, as in theembodiment described above, the execution unit 104 is operated in thefollowing mode in a case in which the frequency at which the differencedegree changes is less than the third threshold value, and is operatedin the non-following mode in a case in which the frequency at which thedifference degree changes is equal to or more than the third thresholdvalue. Therefore, it is possible to reduce the discomfort given to theviewer 28 due to the frequent execution of the specific process ascompared to a case in which the specific process is executed by theexecution unit 104 regardless of the frequency at which the differencedegree between the virtual viewpoint video 46C and the reference video46B1 changes.

In addition, in the embodiment described above, the aspect example hasbeen described in which the specific process is executed by theexecution unit 104 regardless of the posture of the smart device 14, butthe technology of the present disclosure is not limited to this, and thespecific process may be executed or not executed depending on theposture of the smart device 14. In this case, based on the angleinformation measured by the gyro sensor 74, the CPU 88 of the smartdevice 14 detects the posture of the smart device 14. Note that, here,the CPU 88 is an example of a “detection unit (detector)” according tothe technology of the present disclosure. For example, as shown in FIG.33, the CPU 88 outputs posture information indicating the posture of thesmart device 14 to the execution unit 104. The execution unit 104includes a posture determination unit 104F, and the posturedetermination unit 104F determines whether or not the posture indicatedby the posture information input from the CPU 88 is a predeterminedposture. The predetermined posture refers to a posture in which thedisplay 78 is not vertically downward, for example.

In a case in which the posture indicated by the posture information isthe predetermined posture, the posture determination unit 104F outputsexecution instruction information for giving an instruction of theexecution of the specific process to the difference degree calculationunit 104A, the determination unit 104B, and the notification processingunit 104C. In a case in which the execution instruction information isinput to the difference degree calculation unit 104A, the determinationunit 104B, and the notification processing unit 104C, the differencedegree calculation unit 104A, the determination unit 104B, and thenotification processing unit 104C are operated to execute the specificprocess described above. In addition, in a case in which the postureindicated by the posture information is not the predetermined posture,the posture determination unit 104F outputs non-execution instructioninformation for giving an instruction of the non-execution of thespecific process to the difference degree calculation unit 104A, thedetermination unit 104B, and the notification processing unit 104C. In acase in which the non-execution instruction information is input to thedifference degree calculation unit 104A, the determination unit 104B,and the notification processing unit 104C, the difference degreecalculation unit 104A, the determination unit 104B, and the notificationprocessing unit 104C are not operated, and the specific processdescribed above is not executed. Therefore it is possible to execute ornot execute the specific process by changing the posture of the smartdevice 14.

In addition, although the fixed value is adopted as the first thresholdvalue described in the embodiment described above, the technology of thepresent disclosure is not limited to this, and the variable value whichis changed in response to the instructions received by the receptiondevice 52 (see FIG. 2) and/or the reception device 76 may be adopted. Inthis case, for example, as shown in FIG. 34, in a case in which firstthreshold value change instruction information for giving an instructionfor changing the first threshold value is received by the touch panel76A of the smart device 14, the change of the first threshold value maybe executed by the execution unit 104 according to the first thresholdvalue change instruction information. Here, the first threshold valuechange instruction information is an example of “instructioninformation” according to the technology of the present disclosure.

In the example shown in FIG. 34, the execution unit 104 includes aninstruction information reception unit 104G, and the first thresholdvalue change instruction information is transmitted from the smartdevice 14 to the instruction information reception unit 104G in a casein which the first threshold value change instruction information isreceived by the touch panel 76A of the smart device 14. The firstthreshold value change instruction information includes informationindicating the first threshold value or information indicating a degreeof the change in the first threshold value. The instruction informationreception unit 104G receives the first threshold value changeinstruction information to output the received first threshold valuechange instruction information to the determination unit 104B. In a casein which the first threshold value change instruction information isinput to the determination unit 104B, the determination unit 104Bchanges the first threshold value used for comparison with thedifference degree according to the first threshold value changeinstruction information. Therefore, as compared to a case in which thefirst threshold value is always fixed, it is possible to easilydetermine the intention of the viewer 28 or the like for the change inthe first threshold value.

In addition, in the embodiment described above, the aspect example hasbeen described in which the display control process (see FIG. 19), thesetting process (see FIG. 20), and the information processing (see FIGS.21 and 22) are executed by the CPU 58 of the information processingapparatus 12, but the technology of the present disclosure is notlimited to this. At least one of the display control process, thesetting process, or the information processing may be executed by theCPU 88 of the smart device 14. In the following, in a case in which adistinction is not necessary, the display control process, the settingprocess, and the information processing are referred to as an“information processing apparatus side process”.

FIG. 35 shows a configuration example of the smart device 14 in a casein which the information processing apparatus side process is executedby the CPU 88 of the smart device 14. For example, as shown in FIG. 35,the storage 90 stores the information processing apparatus side program.In addition, the CPU 88 executes the information processing by beingoperated as the acquisition unit 102 and the execution unit 104according to the information processing program 60A. The CPU 88 executesthe display control process by being operated as the control unit 106according to the display control program 60B. Further, the CPU 88executes the setting process by being operated as the setting unit 108according to the setting program 60C.

Note that in the example shown in FIG. 35, the smart device 14 is anexample of an “information processing apparatus” according to thetechnology of the present disclosure. Note that in the example shown inFIG. 35, the smart device 14 is described as an example of the“information processing apparatus” according to the technology of thepresent disclosure, instead of the smart device 14, various devicesequipped with an arithmetic device, such as a head up display, a headmounted display, a personal computer and/or a wearable terminal, canalso be adopted as the “information processing apparatus” according tothe technology of the present disclosure.

In addition, in the embodiment described above, the soccer stadium 22has been described as an example, but it is merely an example, and anyplace, such as a baseball stadium, a rugby stadium, a curling stadium,an athletics stadium, a swimming pool, a concert hall, an outdoor musichall, and a theater venue, may be adopted as long as the plurality ofimaging apparatuses and the plurality of sound collection devices 100can be installed.

In addition, in the embodiment described above, the wirelesscommunication method using the base station 20 has been described as anexample, but it is merely an example, and the technology of the presentdisclosure is established even in the wired communication method usingthe cable.

In addition, in the embodiment described above, the unmanned aerialvehicle 27 has been described as an example, but the technology of thepresent disclosure is not limited to this, and the imaging region may beimaged by the imaging apparatus 18 suspended by a wire (for example, aself-propelled imaging apparatus that can move along the wire).

In addition, in the embodiment described above, the computers 50 and 70have been described as an example, but the technology of the presentdisclosure is not limited to theses. For example, instead of thecomputers 50 and/or 70, a device including an ASIC, an FPGA, and/or aPLD may be applied. In addition, instead of the computers 50 and/or 70,a combination of a hardware configuration and a software configurationmay be used.

In addition, in the embodiment described above, the informationprocessing apparatus side program is stored in the storage 60, but thetechnology of the present disclosure is not limited to this, and asshown in FIG. 36, for example, the information processing apparatus sideprogram may be stored in any portable storage medium 500, such as an SSDor a USB memory, which is a non-transitory storage medium. In this case,the information processing apparatus side program stored in the storagemedium 500 is installed in the computer 50, and the CPU 58 executes theinformation processing apparatus side process according to theinformation processing apparatus side program.

In addition, the information processing apparatus side program may bestored in a storage unit of another computer or a server deviceconnected to the computer 50 via a communication network (not shown),and the information processing apparatus side program may be downloadedto the information processing apparatus 12 in response to the request ofthe information processing apparatus 12. In this case, the informationprocessing apparatus side process based on the downloaded informationprocessing apparatus side program is executed by the CPU 58 of thecomputer 50.

In addition, in the embodiment described above, the CPU 58 has beendescribed as an example, but the technology of the present disclosure isnot limited to this, and a GPU may be adopted. In addition, a pluralityof CPUs may be adopted instead of the CPU 58. That is, the informationprocessing apparatus side process may be executed by one processor or aplurality of physically separated processors. In addition, instead ofthe CPU 88, a GPU may be adopted, a plurality of CPUs may be adopted, orone processor or a plurality of physically separated processors may beadopted to execute various processes.

The following various processors can be used as a hardware resource forexecuting the information processing apparatus side process. Examples ofthe processor include a CPU, which is a general-purpose processor thatfunctions as software, that is, the hardware resource for executing theinformation processing apparatus side process according to the program,as described above. In addition, another example of the processorincludes a dedicated electric circuit which is a processor having acircuit configuration specially designed for executing a specificprocess, such as an FPGA, a PLD, or an ASIC. A memory is also built inor connected to each processor, and each processor executes theinformation processing apparatus side process by using the memory.

The hardware resource for executing the information processing apparatusside process may be configured by one of the various processors, or maybe a combination of two or more processors of the same type or differenttypes (for example, a combination of a plurality of FPGAs or acombination of a CPU and an FPGA). In addition, the hardware resourcefor executing the information processing apparatus side process may beone processor.

As an example of configuring the hardware resource with one processor,first, as represented by a computer such as a client computer or aserver, there is a form in which one processor is configured by acombination of one or more CPUs and software, and the processorfunctions as the hardware resource for executing the informationprocessing apparatus side process. Secondly, as represented by SoC,there is an aspect in which a processor that realizes the functions ofthe whole system including a plurality of the hardware resources forexecuting the information processing apparatus side process with one ICchip is used. In this way, the information processing apparatus sideprocess is realized by using one or more of the various processorsdescribed above as the hardware resource.

Further, as the hardware structure of these various processors, morespecifically, an electric circuit in which circuit elements such assemiconductor elements are combined can be used.

In addition, the information processing apparatus side process describedabove is merely an example. Therefore, it is needless to say thatunnecessary steps may be deleted, new steps may be added, or the processorder may be changed within a range that does not deviate from the gist.

The contents described and shown above are the detailed description ofthe parts according to the technology of the present disclosure, and aremerely examples of the technology of the present disclosure. Forexample, the description of the configuration, the function, the action,and the effect above are the description of examples of theconfiguration, the function, the action, and the effect of the partsaccording to the technology of the present disclosure. Accordingly, itis needless to say that unnecessary parts may be deleted, new elementsmay be added, or replacements may be made with respect to the contentsdescribed and shown above within a range that does not deviate from thegist of the technology of the present disclosure. In addition, in orderto avoid complications and facilitate understanding of the partsaccording to the technology of the present disclosure, in the contentsdescribed and shown above, the description of common technical knowledgeand the like that do not particularly require description for enablingthe implementation of the technology of the present disclosure areomitted.

In the present specification, “A and/or B” is synonymous with “at leastone of A or B”. That is, “A and/or B” means that it may be only A, onlyB, or a combination of A and B. In addition, in the presentspecification, in a case in which three or more matters are associatedand expressed by “and/or”, the same concept as “A and/or B” is applied.

All of the documents, the patent applications, and the technicalstandards described in the present specification are incorporated in thepresent specification by referring to the same extent as a case in whichindividual document, patent application, and technical standard arespecifically and individually noted to be incorporated by reference.

Regarding the embodiment described above, the following supplementarynote will be further disclosed.

(Supplementary Note 1)

An information processing apparatus including a processor, and a memorybuilt in or connected to the processor,

-   -   in which the processor acquires a free viewpoint video based on        at least one image among a plurality of images obtained by        imaging an imaging region by a plurality of imaging apparatuses,        and a reference video, the free viewpoint video indicating the        imaging region in a case in which the imaging region is observed        from a specific viewpoint position and a specific visual line        direction, and    -   executes a specific process in a case in which a difference        degree between the acquired free viewpoint video and the        acquired reference video is equal to or more than a first        threshold value.

What is claimed is:
 1. An information processing apparatus comprising: a processor; and a memory built in or connected to the processor, wherein the processor acquires a free viewpoint video based on at least one image among a plurality of images obtained by imaging an imaging region by a plurality of imaging apparatuses, and a reference video, the free viewpoint video indicating the imaging region in a case in which the imaging region is observed from a specific viewpoint position and a specific visual line direction, and executes a specific process in a case in which a difference degree between the acquired free viewpoint video and the acquired reference video is equal to or more than a first threshold value.
 2. The information processing apparatus according to claim 1, wherein the specific process is a process including a notification process of giving notification that the difference degree is equal to or more than the first threshold value.
 3. The information processing apparatus according to claim 2, wherein the notification process is a process of giving at least one of visible notification, audible notification, or tactile notification that the difference degree is equal to or more than the first threshold value.
 4. The information processing apparatus according to claim 1, wherein the specific process is a process including a process of limiting a range in which the viewpoint position is able to be indicated to a viewpoint position range determined as a range in which the difference degree is set to be less than the first threshold value, and limiting a range in which the visual line direction is able to be indicated to a visual line direction range determined as a range in which the difference degree is set to be less than the first threshold value.
 5. The information processing apparatus according to claim 1, wherein the specific process is a process including a change process of performing, on the free viewpoint video change for setting the difference degree to be less than the first threshold value.
 6. The information processing apparatus according to claim 5, wherein the change process is a process of changing the free viewpoint video by changing at least one of the viewpoint position or the visual line direction.
 7. The information processing apparatus according to claim 1, wherein the processor executes the specific process depending on a change in the free viewpoint video.
 8. The information processing apparatus according to claim 7, wherein the processor selectively sets a following mode in which the processor executes the specific process while following the change in the free viewpoint video, and a non-following mode in which the processor does not execute the specific process while following the change in the free viewpoint video.
 9. The information processing apparatus according to claim 8, wherein the processor sets the following mode in a case in which a change amount of the difference degree is less than a second threshold value, and sets the non-following mode in a case in which the change amount is equal to or more than the second threshold value.
 10. The information processing apparatus according to claim 8, wherein the processor sets the following mode in a case in which a frequency at which the difference degree changes is less than a third threshold value, and sets the non-following mode in a case in which the frequency is equal to or more than the third threshold value.
 11. The information processing apparatus according to claim 1, wherein the processor further displays the free viewpoint video on a first display device.
 12. The information processing apparatus according to claim 11, wherein the first display device includes a detector that detects a posture of the first display device, and the processor executes the specific process in a case in which the posture detected by the detector is a predetermined posture, and does not execute the specific process in a case in which the posture detected by the detector is a posture different from the predetermined posture.
 13. The information processing apparatus according to claim 11, wherein the processor displays the reference video on a second display device different from the first display device.
 14. The information processing apparatus according to claim 1, wherein the processor executes change in the first threshold value according to instruction information for giving an instruction for changing the first threshold value received by a reception device in a case in which the instruction information is received by the reception device.
 15. The information processing apparatus according to claim 1, wherein the free viewpoint video is a virtual viewpoint video.
 16. The information processing apparatus according to claim 1, wherein the reference video is a live broadcast video.
 17. The information processing apparatus according to claim 1, wherein the reference video is an image obtained by imaging the imaging region by a reference imaging apparatus.
 18. The information processing apparatus according to claim 1, wherein the viewpoint position and the visual line direction are a viewpoint position and a visual line direction as indicated from an outside.
 19. The information processing apparatus according to claim 1, wherein the specific process is a process that contributes to setting the difference degree to be less than the first threshold value.
 20. An information processing method comprising: acquiring a free viewpoint video based on at least one image among a plurality of images obtained by imaging an imaging region by a plurality of imaging apparatuses, and a reference video, the free viewpoint video indicating the imaging region in a case in which the imaging region is observed from a specific viewpoint position and a specific visual line direction; and executing a specific process in a case in which a difference degree between the acquired free viewpoint video and the acquired reference video is equal to or more than a first threshold value.
 21. A non-transitory computer-readable storage medium storing a program for causing a computer to execute a process comprising: acquiring a free viewpoint video based on at least one image among a plurality of images obtained by imaging an imaging region by a plurality of imaging apparatuses, and a reference video, the free viewpoint video indicating the imaging region in a case in which the imaging region is observed from a specific viewpoint position and a specific visual line direction; and executing a specific process in a case in which a difference degree between the acquired free viewpoint video and the acquired reference video is equal to or more than a first threshold value. 